The use of the simple mail transfer protocol (SMTP) to transmit electronic messages and email has grown to the point where it is now a mission-critical service. Organizations now view it as being as vital to their everyday operations as the telephone.
Two significant challenges have emerged with this reliance on email. The first challenge is how to ensure timely delivery of all messages. Users now expect messages to be delivered within minutes or even seconds. Generally, message protocols, particularly simple message transfer protocol (SMTP), are resilient in ensuring the eventual delivery of messages, but the underlying message transfer agent (MTA) software and its hardware platform do not currently contribute to timely delivery, and may themselves serve to introduce delays into the system by virtue of unreliability.
The second important challenge has arisen from the growing value of each particular message. Email is now being used to transmit much more than simple text. Email is used to transmit information of commercial value (purchase orders, shipping waybills, etc), legal documents (contracts, agreements, etc.) as well as collaborative information like scheduling, appointments, etc.
Organizations and individuals can no longer afford either a delay or loss of messages.
Major components of email systems that interact via SMTP can be broadly categorized as email clients, email servers and email gateways.
Email clients, like Microsoft Outlook®, provided by Microsoft Corp. of Redmond, Wash., are used to read and manage the local storage of messages for an individual user and serve as end-points of the email system. Users are generally responsible for ensuring that the messages that they receive are backed up and stored safely. In many cases, this will be mandated by policy and standard tools will be provided for this purpose.
Email servers, like Microsoft Exchange™, provided by Microsoft Corp. of Redmond, Wash., are used to host email accounts, provide a delivery point for messages for a mail domain so that individual users can read or collect their personal messages, and may also be used for storage. There is a wide range of sophisticated schemes available for ensuring that the contents of email servers are properly protected against accidental loss.
Email gateways are used to manage the flow of email into and out of private networks, which may contain one or more email servers. Gateways may also serve other functions like email security and hygiene, and some gateways may be characterized as email firewalls due to their emphasis on these functions.
One function of email gateways is that they can accept email from external systems via a public network, such as the Internet, queue them locally for processing, and relay them on to internal email servers where they can be accessed by email clients. Internal email clients and servers may also use the email gateway to manage, queue, and relay messages to external email servers and mail gateways.
While awaiting relay, email messages are typically stored in queues on the email gateway disks. Relay may be instantaneous, but may also be delayed if the target email server is unavailable. The target email server may be unreachable due to a network error, may be temporarily down for maintenance, or may be unavailable for many other reasons. The email gateway will typically be configured to keep re-trying delivery for a number of days.
Email gateways can be deployed on systems that provide some measure of protection against failure, including hardware redundancy and clusters for alternative delivery routes.
The volume of email that is now being sent implies that these message queues can become quite large, sometimes numbering in the thousands or ten of thousands of messages. Many major organizations receive 50,000 messages an hour. If the system cannot deliver messages because of an intervening network router or switch malfunctions, the queue may grow very quickly to tens of thousands or more before the administrator is notified.
While the messages are on the queues of the email gateway, they are vulnerable to loss. Any hardware or software failure on the email gateway may cause the message to be irrevocably lost.
If the message is not lost, it may still be “stuck” on a system that cannot deliver the message. For example, the system motherboard may have failed, requiring that the messages in a queue be manually retrieved (for example, from a disk storage device) if they are to be delivered. This may impose an unacceptable delay.
If the message is lost, there will be no way of recovering and retrying the delivery. The system that has used the email gateway to relay both inbound and outbound email considers the message to have been delivered. The sender has recorded the “delivery” of the message, but the recipient never receives it and, if the recipient is not expecting the message, has no reason to alert the sender to this fact.
The email gateway has therefore become a weak link in the chain of delivery between email clients and email servers, and between email servers. The delay or loss may represent significant cost to an organization. Therefore, a solution to this problem is desired.